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ABSTRACT 



The National Household Education Survey (NHES ) is a data 
collection system of the National Center for Education Statistics that is 
designed to provide information on educational issues that can best be 
studied through contacting households rather than educational institutions. 
This report compares the information collected through telephone interviews 
by the NHES on adult education in 1991 and 1995 with that collected through 
the Current Population Survey (CPS) of the Bureau of the Census at 3 -year 
intervals between 1969 and 1984 and in 1992. Included in the study were 
examinations of population coverage and the selection of adults for 
interviews, survey nonresponse, the use of proxy reports of adult education 
participation, and survey context. In the NHES in 1991, 60,314 households 
were screened, and the total number of adult interviews completed was 12,568, 
a figure that included 9,774 adults who participated in adult education 
within the preceding 12 months. In 1992, approximately 57,000 households were 
interviewed for the CPS, a monthly survey of households. Estimates of adult 
education participation provided by the CPS are substantially lower than 
those from the NHES. The 1984 participation estimate from the CPS was 14% and 
the 1992 estimate was 21%. Differences in CPS and NHES estimates range from 
13 to 20%. Examination of methodological, operational, and definitional 
issues that might account for the differences suggest that NHES figures are 
more accurate, although even these figures may underestimate participation. 
Reasons for the discrepancies are discussed. An appendix presents adult 
education participation items from the surveys. (Contains 9 tables and 37 
references.) (SLD) 
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Introduction 

The measurement of participation in adult 
education (AE) activities presents a number of 
definitional and methodological challenges. 
Zemsky and Shapiro (1993) note that estimates 
of job-related training differ "not just widely, 
but wildly," and liken the attempt to provide 
accurate training estimates to "measuring a 
mirage." Estimates of participation in a broad 
range of adult education activities from the 
National Household Education Survey and the 
Current Population Survey also vary (Hill 1987; 
Korb, Chandler, and West 1991; Kim et al. 
1995). The ways in which a given research 
study defines adult education and the methods 
used to sample persons and collect information 
about their educational experiences may have a 
substantial impact on the observed rates of 
participation. 

The purpose of this report is to compare two 
studies that have yielded disparate rates of AE 
participation at several points in time, and to 
examine the definitional and methodological 
factors that may lead to these different results. 
The two studies on which this analysis focuses 
are the National Household Education Survey 
(NHES) and the Current Population Survey 
(CPS). The NHES included adult education 
components in 1991 and 1995. The CPS 
collected information on participation in adult 
education every three years from 1969 through 
1984, and the 1984 participation rate is 
discussed in this report. In addition, the CPS 
included a brief set of questions on adult 
education in the October 1992 educational 
supplement; those items replicated items used to 
estimate the AE participation rate in the 
NHES:91. Included in this study are 
examinations of population coverage and the 
selection of adults for interviews, survey 
nonresponse, the use of proxy reports of adult 
education participation, and survey context. In 
each of these investigations, the appropriate data 
sets are used to examine the specific issue at 
hand, because not all of the selected data sets 
lend themselves to each of the methodological 
factors explored in this research. 
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Before proceeding further, brief overviews of 
each of these data collection systems are 
presented. 

The National Household Education 
Survey 

The National Household Education Survey 
(NHES) is a data collection system of the 
National Center for Education Statistics 
(NCES), which has as its legislative mission the 
collection and publication of data on the 
condition of education in the Nation. The 
NHES is specifically designed to support this 
mission by providing information on those 
educational issues that can be addressed by 
contacting households rather than schools or 
other educational institutions. The NHES 
provides descriptive data on the educational 
activities of the U.S. population and offers 
policy makers, researchers, and educators a 
variety of statistics on the condition of 
education in the U.S. By focusing specifically 
on educational topics of interest, the NHES 
permits the collection of more in-depth 
information than can be collected in 
supplements to existing household surveys. 

The NHES is a telephone survey of the 
noninstitutionalized civilian population of the 
U.S. Households are selected for the survey 
using random digit dialing (RDD) methods, and 
data are collected using computer-assisted 
telephone interviewing (CATI) procedures. 
About 45,000 to 64,000 households are screened 
for each administration, and individuals within 
households who meet predetermined criteria are 
sampled for more detailed or extended 
interviews. The data are weighted to make 
estimates of the entire population. The NHES 
survey for a given year typically consists of a 
Screener, which collects household composition 
and demographic data, and extended interviews 
on two substantive components addressing 
education-related topics. In order to assess data 
item reliability and inform future NHES surveys, 
each administration also includes data quality 
activities, such as a reinterview with a subsample 
of respondents. 
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The primary purpose of the NHES is to conduct 
repeated measurements of the same phenomena 
at different points in time and one-time surveys 
on topics of interest to the Department of 
Education. Throughout its history, the NHES has 
collected data in ways that permit estimates to be 
tracked across time. This includes repeating 
topical components on a rotating basis in order to 
provide comparative data across survey years. In 
addition, each administration of the NHES has 
benefited from experiences with previous cycles, 
resulting in enhancements to the survey 
procedures and content. Thus, while the survey 
affords the opportunity for tracking phenomena 
across time, it is also dynamic in addressing new 
issues and including conceptual and 
methodological refinements. 

Full-scale implementations of the NHES have 
been conducted in 1991, 1993, 1995, and 1996. 
Topics addressed by the NHES:91 were early 
childhood education and adult education. The 
NHES:93 collected information about school 
readiness and school safety and discipline. The 
1991 components were repeated for the 
NHES:95, addressing early childhood program 
participation and adult education. Both 
components underwent substantial redesign to 
incorporate new issues and develop new 
measurement approaches. In the NHES:96, the 
topical components were parent/family 
involvement in education and civic involvement, 
and an expanded screening approach provided 
state-level estimates of public library use by 
household. 

In the NHES:91, 60,314 households were 
screened and the total number of completed adult 
education interviews was 12,568, which included 
interviews with 9,774 adults who participated in 
adult education in the previous 12 months and 
2,794 interviews with adults who did not 
participate. In the NHES:95, of the 45,465 
households with completed screener interviews, 
19,722 adult education interviews were 
completed, including 11,713 AE participants and 
8,009 AE nonparticipants. The NHES:95 also 
contained a methodological experiment to help 
evaluate whether any changes between 1991 and 
1995 were the result of changes in the design of 



the interview or were due to changes in the 
participation rate over time. This methodological 
experiment, called the NHES:95 Splice sample, 
used the interview design of the NHES:91. A 
total of 4,082 Splice sample interviews was 
completed in the NHES:95. More details on the 
NHES:95 data collection and public use data files 
are given in National Household Education 
Survey of 1995: Adult Education Data File 

User ‘s Manual (Collins et al. 1996). 

The Current Population Survey 

The Current Population Survey (CPS) is a 
monthly survey of households conducted by the 
U.S. Bureau of the Census. The main purpose 
of the CPS is to provide estimates of 
employment, unemployment, and other 
characteristics of the labor force. Through the 
use of supplements to the main CPS interview, 
estimates are obtained for other topics of 
interest. Among these are supplements 
pertaining to educational issues. For example, 
the October School Enrollment Supplement 
provides specific information on the enrollment 
status of individuals in the population by 
demographic and socioeconomic characteristics. 
A supplement on participation in adult 
education activities was conducted every three 
years from 1969 through 1984 (Hill 1987). 
Questions on adult education were also included 
in October 1992, and this brief set of questions 
replicated the items used to estimate the 
participation rate in the NHES:91; the same 
questions were used in the NHES:95 AE Splice 
sample interview described above. 

The CPS is a sample of the civilian, 
noninstitutionalized population of the United 
States, and estimates from the survey exclude 
active duty military personnel. The CPS is 
based on a multistage stratified probability 
sample. The first stage involves sampling 
Primary Sampling Units (PSUs) from 
homogeneous strata where the PSUs are 
counties or groups of counties. In the second 
stage a cluster of addresses is sampled from 
each of the sampled PSUs. The persons living 
in the sampled addresses are interviewed in a 
rotation scheme. The addresses are included in 
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the sample for four consecutive months, 
excluded for eight months, then included for 
four more months. Typically, the first interview 
and the fifth interviews are done in person and 
the other interviews are done by telephone, 
where possible. More details on the design of 
the CPS is given in U.S. Department of 
Commerce (1978). 

The total monthly sample size in 1992 was 
approximately 60,000 housing units, of which 
about 57,000 were interviewed. The household 
respondent must be a knowledgeable household 
member aged 14 years or more; this respondent 
provides information for each household 
member. In some cases, individuals are selected 
to respond to supplements about their activities 
or characteristics. However, in some 
supplements other knowledgeable household 
members (proxy respondents) may be allowed 
to respond if the targeted person is not available 
to complete the supplementary interview. In the 
October CPS supplement on adult education, 42 
percent of the responses were by proxies. 
About two-thirds of all CPS interviews are 
conducted by telephone; the remainder are 
conducted by personal interviews. In the 
October CPS:92, the month in which the adult 
education supplement was conducted, 70 
percent of the interviews were conducted by 
telephone.! 

The triennial Adult Education Supplement to the 
CPS (1969-1984) was collected for adults who 
were reported by the household respondent as 
having participated in adult education activities 
in the previous 12 months. Persons who were 
not reported as participants were not asked to 
complete the AE supplement. Furthermore, the 
sampled adult became ineligible if that adult 
said they had not participated in adult education 
in the past year. In the AE supplement itself, 
adults were asked about their participation in a 
variety of educational activities. Those who had 
participated in activities other than, or in 
addition to, full-time enrollment were asked to 



! Special tabulation from the 1992 Current Population 
Survey. 
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describe the four most recent courses in which 
they participated (Hill 1987). 

As noted above, in October 1992, data from 
adults about their participation in AE activities 
were again collected in the education 
supplement to the CPS. Adults were defined as 
civilian, noninstitutionalized persons 16 years or 
older who were not enrolled full-time in 
elementary or secondary school. In this survey, 
the participation items were those used in the 
NHES:91 AE component. The household 
respondent answered for all adult household 
members in most cases, so some data are proxy 
responses. The question only included the items 
defining participation and excluded the more 
detailed information on adult education 
collected in the NHES. All adults were eligible 
for this supplement without asking the 
household respondent any preliminary screening 
questions about the adult education participation 
of other adults in the household. 

What is Adult Education? 

The measurement of adult education 
participation is problematic in part because the 
learning activities that fall within the realm of 
adult education can be defined in a number of 
ways. Darkenwald and Merriam (1982) define 
adult education as participation in systematic 
learning activities for the purpose of acquiring 
new knowledge or skills by persons who have 
assumed adult social roles. This definition 
potentially encompasses a large range of 
activities that may include basic skills 
education, activities leading to academic 
credentials, work-related courses or training, 
and courses or programs taken for personal 
development, general interest, or recreation. In 
addition, the method of participation may 
include traditional schooling; participation in 
classes, workshops or seminars; self-directed 
learning through correspondence courses or 
computer- or video-based tutorials; or courses 
offered by television or on video or audio tapes 
(Knowles 1980). 
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Some studies of adult education focus 
specifically on basic skills education (Anderson 
and Darkenwald 1979), while others examine 
only work-related training (Frazis, Herz, and 
Horrigan 1995). Still others look at educational 
activities that include participation in structured, 
formal learning activities with an instructor, 
excluding self-teaching activities using books, 
television, video tapes, or audio tapes. 

The estimate of the percentage of adults who 
participate in adult education activities, (i.e., 
participation rates, may differ by whether they 
include full-time or part-time activities, or both, 
and by whether they include courses leading to 
college degrees or other postsecondary 
credentials)(Hill 1987; Korb, Chandler, and West 
1991; Kim et al. 1995). The broadest definition 
includes academic, vocational, work-related, and 
a vocational courses taken in classroom settings, 
in informal settings, and those involving the use 
of television, radio, video or audio tapes, or self- 
directed learning. The studies under investigation 
here, the NHES and the CPS, each took broad 
and inclusive approaches to defining adult 
education. Details about how adult education 
was defined and how those definitions affected 
estimates of participation rates are discussed in 
later sections of this report. 

First, some estimates of AE participation from 
the NHES (1991 and 1995) and CPS (1984 and 
1992) components on adult education are 
presented. Within the context of these disparate 
participation rates, subsequent sections of the 
report examine the definitions of adult education 
used and other methodological and operational 
differences that could account for some of the 
differences in the survey estimates. 

Adult Education Participation 
Rates 

Table 1 shows the total number of adults and the 
adult education participation rates for five data 
sources, in chronological order: 

■ The 1984 Adult Education Supplement 
to the CPS (CPS: 84); 



■ The 1991 National Household Education 
Survey (NHES:91); 

■ The 1992 Adult Education Supplement 
to the CPS (CPS:92); 

■ The 1995 National Household Education 
Survey (NFES:95); and 

■ A special methodological study included 
in the NHES:95 in which a Splice sample 
(a separate, supplementary sample) was 
used to examine whether any differences 
in observed NHES rates were due to 
questionnaire changes made to the AE 
questionnaire between the 1991 and 1995 
surveys (NHES:95 Splice). 

Table 1 shows participation rates based on two 
different sets of AE activities: participation 

rates in any adult education activity and 
participation rates in adult education activity 
excluding full-time credential programs. The 
second type of participation rate includes 
participation in credential programs on a part- 
time basis or in activities other than credential 
programs on a part-time or full-time basis. The 
AE participation rates from these five data 
sources, in general, show a trend toward greater 
participation over time. This observed growth 
is consistent with the findings of the triennial 
adult education supplement to the CPS 
conducted from 1969 through 1984, which 
reflected modest but significant growth over the 
15-year period (Hill 1987). However, the 
estimates from the CPS: 84 and the CPS:92 are 
markedly lower than those from the NHES:91 
and the NHES:95. One might be tempted to 
attribute the differences between the 1984 CPS 
participation rate of 14 percent and the NHES 
rates of 33 and 40 percent as reflecting 
continued growth in AE participation. 
However, the CPS:92 estimated rate of 20.5 
percent, when considered in this chronological 
context, suggests that differences are not due to 
growth alone, and that there are aspects of the 
measurement of AE participation that are 
contributing to these disparate estimates. 



Table 1. — Estimated number of adults 1 and percentage of those participating in adult education from the 
Current Population Survey and the National Household Education Survey: 1984-1995 



Participation Rate 


CPS:84 


NHES:91 


CPS:92 


NHES:95 


NHES:95 

Splice 


Number of adults (in thousands) 


172,583 


181,800 


169,772 


189,576 


189,912 


Any adult education activity 




37.9% (0.7) 


24.0% (2.5) 


44.3% (0.5) 


44.6% (1.0) 


Adult education activity excluding 
full-time degree programs 


14.0% (0.1) 


33.0% (0.7) 


20.5% (0.5) 


40.2% (0.5) 


40.0% (1.0) 



1 Adults include persons age 16 and older not enrolled in elementary or secondary school and not on activity duty in the Armed 
Forces. Data from the 1984 survey (Hill 1987) include persons age 17 and older. 

^Not available from the published report (Hill 1987). 

NOTE: Standard error in parentheses. 

SOURCES: U.S. Department of Commerce, Bureau of Census, Current Population Survey (CPS), 1984 Adult Education 

Supplement, 1992 Adult Education Supplement; U.S. Department of Education, National Center for Education Statistics, National 
Household Education Survey (NHES), spring 1991, spring 1995, spring 1995 Splice Sample Interview. 



The specific types of adult education that were 
asked about in the NHES and CPS were 
examined in order to identify the types of 
activities that contributed to the differences in 
overall participation estimates. Table 2 shows 
the participation rates for the specific types of 
full-time and part-time AE activities. The 
CPS:92 estimates are significantly lower than the 
NHES:91 estimates for full-time participation in 
literacy programs and "other educational 
programs." In addition, the CPS:92 estimates of 
full-time participation are significantly lower 
than NHES:95 Splice estimates for associate's 
degree programs and ESL programs. More 
differences in participation rates are observed 
when part-time AE activities are examined. The 
CPS:92 estimates are significantly lower than the 
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NHES:91 and NHES:95 Splice estimates for part- 
time continuing education, mail or television 
courses, private instruction or tutoring, and 
employer training. In addition, the CPS:92 
estimates are lower than NHES:95 Splice 
estimates for part-time basic skills education and 
"other" part-time educational activities. 

These findings indicate that the reporting of 
activities outside of formal schooling is more 
problematic than reports of formal schooling. 
This is consistent with the results of the NHES:95 
reinterview program and a response bias study 
examining reports of AE participation in the 
NHES:95 AE interview (Brick et al. 1996a, Brick 
et al. 1996b). 




Table 2. — Comparisons of AE participation rates from NHES:91, CPS:92 , and NHES:95 Splice Sample 



Participation items 


Participation rates in 
NHES:91 


Participation rates in 
CPS:92 


Participation rates in 
NHES:95 Splice 


Full-time educational programs 








An elementary, junior high, or high school 


11.7% (1.9) 


13.3% (1.4) 


13.6% (3.9) 


An associate's degree program 


23.1 (1.2) 


18.6 (1.7) 


29.4 (2.9) 


A bachelor's or advanced degree program 


63.4 (1.6) 


61.1 (3.0) 


58.7 (2.9) 


A vocational/occupational training program 


17.2 (1.1) 


15.2 (1.5) 


18.8 (2.6) 


An adult literacy or basic skills program 


3.6 (.6) 


1.2 (.4) 


5.6 (2.1) 


An English as a Second Language program 


2.3 (.4) 


1.1 (.4) 


4.4 (1.1) 


Other educational program 


2.4 (.6) 


5.0 (.9) 


2.1 (1.0) 


Part-time educational programs 








Continuing education 


12.0% ( ,4) 


6.4% (1.0) 


13. 8% (.5) 


Courses by mail, TV, radio 


1.6 (.1) 


•8 ( .3) 


■ ■ 2.7 (.4) 


Private instruction 


2.9 (.2) 


.9 (.4) 


4.5 (.5) 


Training given by employer 


22.5 (.6) 


11.1 (.3) 


29.9 (1.0) 


Instruction in basic skills 


1.3 (.1) 


.6 (.3) 


3.5 (.4) 


Instruction in ESL 


.6 (.5) 


. .6 (.3) 


1.6 (.3) 


Other educational activity 


.6 (.1) 


1.7 (.5) 


1 (!) 



NOTE: Standard error in parentheses. 

SOURCES: U.S. Department of Commerce, Bureau of Census, Current Population Survey (CPS) 1992 Adult Education 

Supplement; U.S. Department of Education, National Center for Education Statistics, National Household Education Survey 
(NHES), spring 1991, spring 1995 Splice Sample Interview. 



In the sections that follow, we discuss several 
aspects of the methods used in these five surveys 
and their potential effects on the observed 
participation rates. The specific aspects that are 
considered include; 

■ population coverage and selection 

methods; 

■ nonresponse; 

■ proxy reporting of adult education 

participation; and 
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■ issues related to the context of the 

surveys. 

Population Coverage and Selection 
Methods 

In this section, factors associated with survey 
coverage of households and persons within 
households and the selection of adults within 
households for adult education data collection 
are discussed. The estimates for both the NHES 
and the CPS include all civilian, 

6 
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noninstitutionalized adults, age 16 and older, 2 
in the 50 states and the District of Columbia 
who were not enrolled in elementary or 
secondary school. However, there are 
differences between the CPS and the NHES, and 
differences between years within each survey, in 
the sampling of households and the selection of 
adults for the collection of information on AE 
participation. The NHES is an RDD telephone 
survey and includes only telephone households, 
whereas the CPS is a sample of both telephone 
and nontelephone households. Thus, the CPS 
covers adults living in households without 
telephones who are not sampled in the NHES. 

Population coverage is not related to the coverage 
of households alone. The coverage of persons 
within households is also of concern. An 
investigation of within household coverage, 
comparing in-person and telephone surveys, was 
conducted by Maklan and Waksberg (1988). 
They found that within-household coverage is no 
more problematic in telephone surveys than in- 
person household surveys. 

In addition to population coverage, there are also 
variations in the selection of adults within the 
households. In the CPS:84, only those adults 
identified as participants by the household 
respondent who completed the CPS AE screening 
items were sampled for the AE supplement. In 
the CPS:92, participation data were collected for 
all adults, but proxy reporting was used for most 
adults other than the household respondent. In 
the NHES:91 and NHES:95, adults were 
classified as participants and nonparticipants at 
the screening stage and both types of adults were 
sampled for AE extended interviews, but at 
different sampling rates. In the NHES:91, 
participants were sampled at rates up to 14 times 



^The 1984 CPS estimates are for adults age 17 and older. 
The NHES:91 sample includes only 17 eligible 16-year- 
olds out of a total adult sample of 12,568, and the 
NHES:95 sample includes only 13 eligible 16-year-olds 
out of a sample of 19,722. The NHES:95 participation rate 
is 40 percent regardless of whether 16-year-olds are 
included or not. Thus, the absence of 16-year-olds from 
the 1984 estimates is not problematic in considering 
differences in rates. 



greater than nonparticipants. In the NHES:95, 
both participation status and high school 
completion were considered in sampling, but 
there was less variation in the sampling rates with 
rates for participants and nonparticipants varying 
by a factor of five at a maximum. 

Below, the implications of differences between 
the CPS and the NHES in terms of the coverage 
of households and the potential for lack of 
coverage of nontelephone households in the 
NHES as explanatory factors in the estimates of 
AE participation are examined. Following this, 
the implications of differences in selecting adults 
within households are addressed. 

Telephone Coverage and Estimates of 
Participation 

The telephone coverage of adults in the 
NHES:95 is discussed in some detail in Brick 
(1996). Some of the important findings of that 
work are summarized here. Approximately 94 
percent of all persons live in households with 
telephones, according to data from the March 
1992 CPS. The percentage of persons who live 
in households with telephones varies somewhat 
by characteristics of the populations considered. 
For example, while 95 percent of all adults age 
16 years and older live in telephone households, 
only 87 percent of black adults and 88 percent 
of Hispanic adults live in telephone households, 
according to these CPS estimates. The 
difference in telephone coverage rates by 
characteristics of the population is one of the 
factors that leads to biases in statistics based on 
data collected from persons in telephone 
households only. 

The term bias has a specific technical definition 
in this context; bias is the expected difference 
between the estimates from the survey and the 
actual population value. For example, if all 
telephone households were included in the 
survey and responded to the interview, the 
difference between the estimate from the survey 
and the actual population value (which includes 
the responses of persons living in nontelephone 
households) is the bias due to incomplete 
coverage. Since the NHES is based on a 
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sample, the bias is defined as the expected or 
average value of this difference over all possible 
samples. 

Coverage bias, the bias due to failure to give all 
persons a chance to be selected in the sample, 
can be substantial when two conditions hold. 
First, the differences between the characteristics 
in the covered population and the uncovered 
population must be relatively large. For 
example, consider estimating the percentage of 
persons enrolled in a given type of program. If 
the percentage enrolled is nearly . identical in 
both the covered and uncovered populations, 
then the bias for the estimate will be negligible. 

Second, the proportion , of the population not 
covered by the survey must be large compared 
to the size of the estimates. If only 2 percent of 
the population is not covered, estimates that 
comprise 20 or 30 percent of the population will 
not be greatly affected, even if the differences in 
the characteristics between the covered and 
uncovered populations are relatively large. It is 
important to realize that this condition requires 
that the proportion not covered must be large 
relative to the size of the estimates. If the 
estimate is for a small domain or subgroup, then 
even a small undercoverage problem can result 
in important biases if the differences in the 
characteristic between the covered and 
uncovered populations are large. Statistics for 
dropouts from high school, a small subgroup, 
suffer from this problem (Brick, Burke, and 
West 1992). 

The bias of an estimate can be expressed 
mathematically to show the relationships 
between the bias and the two factors discussed 
above. The bias is given by 

Bias (Y t ) =P n {E(Y,-Y n )} ( 1 ) 

where Y, is the estimated characteristic based 
on the telephone households only, P n is the 
proportion of nontelephone households, Y n is 
the estimated characteristic based on the 
nontelephone households, and E is the 



expectation operator for averaging over all 
possible samples. 

The undercoverage rate or proportion of adults 
in households without telephones, can be 
estimated directly from the CPS since it covers 
both telephone and nontelephone households. 
For adults eligible for the AE component of the 
NHES:91 and the NHES:95 (civilians who are 
16 years old or older and not currently enrolled 
in elementary or secondary school), 5.6 percent 
lived in nontelephone households, according to 
the October 1992 CPS. 

The differences in the characteristics of persons 
in telephone and nontelephone households have 
been explored for a number of topics by 
different authors. In general, these studies have 
shown that having a telephone is highly related 
to socioeconomic status and lifestyles (Smith 
1990). Thomberry and Massey (1988) assessed 
estimates of health characteristics and found 
many health and health-related characteristics of 
persons in nontelephone households were 
significantly different from those of persons in 
telephone households. Brick, Burke, and West 
(1992), Brick (1992), and Brick (1996) studied a 
variety of estimates for education statistics. 
They found the differences between persons in 
telephone and nontelephone households for 
enrollment statistics were typically smaller than 
those reported by Thomberry and Massey 
(1988). However, for some statistics, such as 
those for high school dropouts, the differences 
were very large. 

In order to examine the impact that excluding 
nontelephone households in the NHES could 
have on differences in estimates of participation 
in adult education, the adult education 
supplement to the October 1992 CPS was used 
to estimate the differences in the characteristics 
of persons in telephone and nontelephone 
households. These differences and the estimates 
of the percent of adults not covered can then be 
used in equation (1) to estimate the bias. 
However, this may overestimate the bias in the 
NHES because some weighting adjustments are 
made in the NHES to reduce the coverage bias. 
In particular, the NHES estimates are raked to 



population totals of adults living in both 
telephone and nontelephone households.^ By 
applying the NHES raking adjustment 
procedures to CPS data for telephone 
households, the amount of residual bias can be 
examined after the weighting adjustments. 

Percentage distributions for selected 
characteristics of adults were tabulated from the 
October 1992 CPS supplement (table 3). The 
tabulation for adults was limited to those adults 
meeting the eligibility criteria for the AE 
components of the NHES, that is, civilians who 
are 16 years old or older and not currently 
enrolled in elementary or secondary school. 
The first three columns of table 3 show the 
estimated percentage distributions for persons in 
telephone households, persons in nontelephone 
households, and persons in all households. The 
fourth column in the table is the estimated 
coverage bias, or the difference between the 
estimate for persons in telephone households 
and the estimate for persons in all households; 
this is the algebraic equivalent of the bias given 
by equation (1). 

The estimates in table 3 show that while the 
characteristics of adults living in telephone and 
nontelephone households may be very different, 
the resulting biases are generally small because 
the undercoverage rates ( P n ) are relatively low. 
Thus, even though adults in telephone 
households were twice as likely as those in 
nontelephone households to have been involved 
in any activities in the last year (24.8 percent 
and 11.8 percent, respectively, from the row 



J Raking is an iterative procedure that ensures that survey 
weights sum to known population totals. It is a calibration 
procedure and is closely related to poststratification. 



labeled “Involved in adult education activity in 
the past year” in table 3), the bias in the estimate 
due to undercoverage for this statistic is only 0.6 
percent (the estimate for all adults, 24.2 percent, 
and the estimate for any adults in telephone 
households is 24.8 percent). Only 5 of the 
statistics in the table have estimated biases 
greater than 0.5 percent. 

Due to the potential biases resulting from 
undercoverage, the standard practice in the 
NHES is to make statistical adjustments of 
survey weights to compensate, to the extent 
possible, for undercoverage. The NHES 
adjustments that are specifically developed to 
compensate for the undercoverage are raking or 
poststratification to known control totals that 
contain counts of persons living in both 
telephone and nontelephone households. The 
goal of these adjustments is to make the 
estimates from the survey consistent with 
known totals (referred to as control totals), to 
partially correct for undercoverage bias, and to 
reduce the variance of the estimates. 

Control totals of the number of persons in both 
telephone and nontelephone households were 
first produced from the CPS file for adults 
eligible for the AE interview. The weights for 
the CPS respondents from telephone households 
were then raked to these control totals to 
produce adjusted weights that summed to the 
total number of persons in both telephone and 
nontelephone households. The responses from 
persons in telephone households were used with 
these adjusted weights to produce adjusted 
estimates. More details are given in Brick 
(1996). 
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Table 3. — Estimated percent of persons by telephone status and estimated coverage bias for adults 



Characteristic 


Telephone 

households 


Non- 

telephone 

households 


All 

households 


Coverage 

bias 


Adjusted 

telephone 

households 


Adjusted 

coverage 

bias 


Highest education 














Less than 12th 


17.2 


40.8 


18.6 


-1.4 


17.9 


-0.7 


12th grade 


36.6 


36.7 


36.6 


0.0 


36.5 


-0.1 


1 or 2 years college 


18.2 


9.8 


17.7 


0.5 


18.0 


0.3 


3-4 years of college 


15.0 


3.4 


14.5 


0.5 


14.7 


0.2 


More than 4 years of college 


7.6 


1.2 


7.3 


0.3 


7.4 


0.1 


Persons aged 15 to 24 years 














High school graduate 


88.1 


53.7 


85.4 


2.7 


87.6 


2.2 


Complete high school by equivalency test 


4.8 


8.7 


5.1 


-0.3 


4.9 


-0.2 


Speak language other than English at home 


13.5 


19.6 


14.0 


-0.5 


14.0 


0.0 


Do not speak English well 


19.4 


39.8 


21.6 


-2.2 


20.1 


-1.5 


Ever had condition affecting ability to learn 


3.2 


3.3 


3.2 


0.0 


3.5 


0.3 


Of those in college, enrolled full-time 


65.1 


69.7 


65.2 


-0.1 


65.5 


0.3 


Now taking business, vocational, technical, 


2.3 


1.8 


2.2 


0.1 


2.3 


0.1 


Reported for person 














Self 


52.6 


60.8 


53.0 


-0.4 


52.8 


-0.2 


Parent 


7.7 


4.2 


7.6 


0.1 


7.8 


0.2 


Spouse 


23.8 


13.9 


23.3 


0.5 


23.4 


0.1 


Other relative 


5.6 


7.3 


5.7 


-0.1 


5.6 


-0.1 


Nonrelative 


2.7 


6.4 


2.9 


-0.2 


2.8 


-0.1 


Involved in adult education activity in last year 














Any activity 


24.8 


11.8 


24.2 . 


0.6 


24.8 


0.6 


Any activity, excluding full-time college 


20.0 


9.9 


19.6 


0.4 


19.9 


0.3 


Any activity, excluding college 


17.6 


8.1 


17.2 


0.4 


17.5 


0.3 


Enrolled full-time in last year 


6.1 


4.4 


6.0 


0.1 


6.3 


0.3 


In elementary or high school program 


12.8 


13.5 


12.8 


0.0 


12.7 


-0.1 


In associated degree program 


17.9 


21.7 


18.1 


-0.2 


18.0 


-0.1 


In bachelor’s or advanced program 


60.5 


31.2 


59.5 


1.0 


60.2 


0.7 


In vocation or occupational program 


14.2 


26.8 


14.7 


-0.5 


14.4 


-0.3 


In literacy or ABE program 


0.9 


5.1 


1.1 


-0.2 


1.0 


-0.1 


In ESL program 


1.0 


2.1 


1.0 


0.0 


1.1 


0.1 


In other program 


4.8 


7.0 


4.8 


0.0 


4.8 


0.0 


Enrolled part-time in college in last year 


3.9 


1.8 


3.8 


0.1 


3.9 


0.1 


In other continuing ed or noncredit program 


6.2 


2.1 


6.0 


0.2 


6.1 


0.1 


In mail, TV, radio, or newspaper courses 


0.8 


0.4 


0.8 


0.0 


0.8 


0.0 


In private instruction 


0.9 


0.4 


0.9 


0.0 


0.9 


0.0 


In program by employer, union, community 














organization 


10.8 


3.5 


10.5 


0.3 


10.7 


0.2 


In basic math or English 


0.6 


1.0 


0.6 


0.0 


0.6 


0.0 


In ESL 


0.5 


0.7 


0.5 


0.0 


0.5 


0.0 


In other organized activity 


1.6 


0.8 


1.6 


0.0 


1.6 


0.0 
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Table 3. — Estimated percent of persons by telephone status and estimated coverage bias for adults — 
Continued 



Characteristic 


Telephone 

households 


Non- 

telephone 

households 


All 

households 


Coverage 

bias 


Adjusted 

telephone 

households 


Adjusted 

overage 

bias 


Involved in AE activity in last year, by race 
Hispanic adults 














Any activity 


20.2 


11.5 


19.2 


1.0 


20.0 


0.8 


Black, non-Hispanic adults 














Any activity 


19.3 


11.2 


18.3 


1.0 


19.1 


0.8 


Any activity, excluding full-time college 


14.5 


8.7 


13.8 


0.7 


14.3 


0.5 


Any activity, excluding college 


12.4 


6.4 


11.6 


0.7 


12.2 


0.5 


Nonblack, non-Hispanic adults 














Any activity 


25.9 


12.3 


25.4 


0.4 


26.0 


0.5 


Any activity, excluding full-time college 


21.0 


10.3 


20.7 


0.3 


21.0 


0.3 


Any activity, excluding college 


18.6 


8.9 


18.3 


0.3 


18.6 


0.3 



NOTE: In this analyses, persons with missing values on participation variables are treated as nonparticipants. 
SOURCE: Special tabulations from the 1992 Current Population Survey (CPS). 



The adjusted estimates are shown in the next to 
last column in table 3. The adjusted bias in 
these statistics is given in the last column, where 
the bias is the difference between the adjusted 
estimate for those in telephone households and 
the estimate from all households. As before, a 
negative coverage bias indicates that the 
estimate is smaller than the estimate based on 
all households. 

These estimates indicate that telephone 
coverage bias accounts for at most a very minor 
component of the differences in the observed 
adult education participation rates for the NHES 
AE components and the CPS AE supplements. 
In particular, the coverage bias for the 
percentage of adults participating in any adult 
education excluding full-time college is 
estimated to be only 0.6 percent (in bold in table 
3). Thus, the percentage of the differences in 
the rates attributable to coverage bias is very 
small, especially when compared to the very 
large differences in the estimates from the CPS 
and NHES AE surveys. 



Selection of Adults 

Although the major focus is on differences 
between the CPS:92 and the NHES estimates, 
the differences between the CPS:84 and the 
NHES:91 estimates initiated this type of 
investigation and do have some implications for 
the comparison of the CPS:92 and the NHES 
differences. A factor that might account for 
some of the difference between the 1984 
estimates and later estimates is the variation in 
procedures for selecting adults within 
households, that is, person-level sampling 
procedures. In both the NHES:91 and the 
CPS:84, a knowledgeable adult member of the 
household was asked a set of screening items 
about the adult education participation of each 
adult in the household; a single screening item 
was used for the NHES:95. In the CPS:84, the 
AE supplement was administered only to those 
adults who were classified at the screening stage 
as participating in adult education (Hill 1987). 
In the CPS:92, information was collected about 
all adults, but proxy reporting was permitted. 
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The CPS:92 can also be considered a screening 
interview like the CPS:84, but without a follow- 
up interview with any of the adults. Both these 
approaches are considered later. In the 
NHES:91 and NHES:95, adults screened as 
participants and as nonparticipants were 
sampled for adult education interviews with 
varying probabilities. For the Splice sample of 
adults in the NHES:95, participation status was 
not a factor in either the screening or the 
sampling procedures. 

In some cases, the household respondent may 
have answered the screening items incorrectly 
for other members of the household because of 
lack of knowledge or because of recall error. 
Because nonparticipants were not selected for 
the CPS:84 AE supplement, these adults did not 
have an opportunity to report on their own 
activities, if any. Some screener respondents 
who reported that they did not participate might 
have recalled activities while completing the 
full interview if they had been given an 
opportunity to respond. This would have 
resulted in the adults being reclassified as 
participants. 

The results of the NHES:91 and the NHES:95 
support the notion that the estimated 
participation rate is reduced by excluding adults 
from the supplement based on participation 
status as reported in screening questions. In the 
NHES, both participants and nonparticipants 
were reclassified in terms of their participation 
status based on their responses at the extended 
interview level. We refer to the percentage 
change in status as the "switching rate." 

In the NHES:91, about 11 percent of those 
sampled as AE nonparticipants who completed 
extended interviews were found to be AE 
participants and about 14 percent of persons 
sampled as participants who completed 
extended interviewers were identified as 
nonparticipants. In the CPS:84, the persons 
who were identified as nonparticipants were not 
sampled, so they had no chance to switch to 
participants in the extended interview. This 
scheme results in a lower estimate of the 
participation rate. Both weighted and 



unweighted estimates show that self-reporters 
were more likely to remain in the same status 
from the screener to the extended interview, 
whether they were initially identified as 
participants or nonparticipants. 

Higher percentages of sampled adults were 
"switchers" in the NHES:95 (table 4). Of those 
sampled as participants, 17 percent were found 
to be nonparticipants. Of those sampled as 
nonparticipants, 23 percent were found to be 
participants. This higher rate of change is most 
likely related to a change in the screening 
question. In the NHES:95, a single brief 
screening question was used compared with a 
longer list of types of AE activities used in the 
NHES:91. This change in the screening 
procedure was made as a result of objections 
from screener respondents in the NHES:91 to 
having to respond to the full list of activities 
repeatedly for each adult in the household. 
Because both participants and nonparticipants 
were sampled in the NHES collections, the loss 
of accuracy associated with the single screening 
item was not considered a major issue, 
especially in light of the opportunity to reduce 
respondent annoyance that might otherwise lead 
to nonparticipation in the study. 

The NHES:95 provides some additional insight 
on the "switching" phenomenon. Table 4 gives 
both weighted and unweighted estimates of 
switching, by the type of switching and the 
screener respondent (self or other). Not 
surprisingly, the extent to which switching 
occurred in the NHES:95 was associated with 
whether the sampled adult was the screener 
respondent or whether another adult in the 
household had answered the screening 
participation question about the sampled adult, 
with the switching rate greater for other 
household responders than for self responders. 

The observed switching of status from the 
screening level to the extended interview level 
suggests that the CPS: 84 participation rate 
would have been higher if persons who were 
initially identified by the household respondent 
as nonparticipants had an opportunity to report 
for themselves. Applying the NHES:91 
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switching rate to the CPS:84, the participation 
rate for the CPS: 84 would have been about 23 
percent. This increase of 9 percent from 14 to 
23 percent results when the estimated 86 percent 
of nonparticipants in the CPS: 84 are assumed to 
be switchers at the same rate as found in the 
NHES:91 (11 percent times 86 percent = 9 
percent). The estimate of 23 percent is still 
substantially below the NHES:91 estimate of 32 
percent, but accounts for a very large percentage 
increase in the participation rate. 

Evidence from the CPS:92 indicates that the 
"switching" phenomenon alone does not account 
for the differences between the NHES and CPS 
estimates. As we saw earlier, the CPS:92 
estimate is 20.5 percent if no adjustments are 
made to the estimates. If the NHES:91 
switching rates are applied to the CPS:92 
estimates, the adjusted estimate is 26.4 percent 
(20.5 percent times 86 percent plus 79.5 percent 
times 1 1 percent =26.4 percent). The difference 



between the adjusted estimate of 23 percent for 
the CPS:84 and the adjusted estimate of 26.4 
percent suggests there was not a substantial 
increase in participation rates in the 8 years 
between the CPS surveys. 

A different approach to adjusting the CPS:92 is 
to model the reporting of those who did not 
report for themselves. This proxy adjustment is 
discussed later, but it leads to an adjusted 
CPS:92 estimate of participation of 25 percent, 
which is very similar to the switching adjusted 
estimate. Both approaches to adjusting the 
CPS:92 reveal little substantial increase in adult 
education participation between 1984 and 1992. 
This contradicts the common impression that 
there has been an increase during this period. 
These results raise doubts about the validity of 
the estimate, especially the unadjusted estimate, 
from the CPS:92, even in comparison to the 
CPS:84 estimates, which have other 
methodological problems. 



Table 4. — Number and percent of sampled adults whose participation status changed from screener to 
extended interview completion by Screener respondent: NHES.95 





Estimates 


Sample 


size 


Number 
(in thousands) 


Percent 

switching 


Standard 

error 


Number 


Percent 

switching 


All adults 












Sampled as participants 


68,971 


16 


.38 


11,890 


17 


Sampled as nonparticipants 


120,605 


22 


.58 


7,832 


23 


Screens respondents (self) 












Sampled as participants 


39,684 


11 


.41 


7,029 


11 


Sampled as nonparticipants 


73,979 


19 


.68 


4,778 


21 


Not screener respondent (other) 












Sampled as participants 


29,287 


24 


.65 


4,861 


25 


Sampled as nonparticipants 


46,626 


26 


.80 


3,084 


27 



SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Survey (NHES), 
spring 1995. 



Survey Nonresponse 

Just as undercoverage can result in biases in the 
estimates, nonresponse, or the failure to obtain a 



response from a sampled unit, may also cause 
the estimates from a survey to differ from the 
population values (Kalton and Kasprzyk 1986; 
Lessler and Kalsbeek 1992). Both the NHES 
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and the CPS suffer from nonresponse, although 
the rates of responding are different in the 
surveys. The potential impact of nonresponse 
on the differences in participation rate estimates 
in the surveys is investigated below. 

Nonresponse is often classified as either unit 
nonresponse or item nonresponse. Unit 
nonresponse occurs when all or nearly all the 
data from the sampled unit (person, household) 
can not be obtained. Unit nonresponse may 
happen for several reasons, such as being unable 
to contact the unit, the respondent refusing, or 
the inability to complete the interview in the 
data collection time/period. Item nonresponse 
occurs when most of the data are obtained, but 
some questions are not answered. As with unit 
nonresponse, there may be many reasons for 
item nonresponse, such as the respondent not 
wanting to answer a particular question like 
income, interviewer error in not asking the 
question, or inconsistent responses to items that 
are detected in later editing. Because item 
nonresponse is generally small compared to unit 
nonresponse in both the NHES and CPS, unit 
nonresponse is the focus of this examination of 
nonresponse as a potential source of differences 
between the survey estimates. However, those 
cases for which all AE items are missing in the 
CPS:92 (about 5.2 percent) are treated as unit 
nonresponse for this analysis. 

The bias due to nonresponse can be expressed in 
the same way as the bias due to undercoverage. 
The bias of the estimate due to nonresponse can 
be written as equation (1), by simply changing 
the subscript t (for telephone households) to r 
(for respondents), and the subscript n now 
indicates nonrespondents while in equation (1) 
it referred to nontelephone households. Thus, 
the nonresponse bias equation is 

Bias(y r ) = P n {E(y r - y n )} . (2) 

As with coverage, this bias is a function of both 
the nonresponse rate ( P n ) and the difference in 
characteristics of the respondents and the 
nonrespondents ( y r — y n ) . As the nonresponse 
rate becomes larger, the bias increases even if 



the difference between the characteristics for 
respondents and nonrespondents remains 
constant. 

The response rates for the NHES:91 and the 
NHES:95 AE components were 69 percent and 
59 percent respectively, while for the NHES:95 
Splice sample the response rate was 64 percent. 
These response rates are weighted by the 
appropriate probabilities of selection and 
account for all levels of unit nonresponse 
(Screener and extended interviews). The 
National Household Education Survey of 1995: 
Adult Education Data File User ’s Manual 
(Collins et al. 1996) contains more information 
on the response rates for the NHES. 

The data from CPS are also subject to 
nonresponse at two levels: failure to complete 
the core interview and failure to complete the 
supplement. The response rate to the core 
interview for the CPS is about 95.6 percent 
(U.S. Department of Commerce 1993). The 
percent of adults with completed AE 
supplement data is 94.8 percent. This yields a 
net response rate of 90.6 percent for the CPS:92 
AE data. This response rate will be assumed for 
the adult education supplement for purposes of 
this assessment. 

The second component of nonresponse bias is 
the differences in characteristics between the 
respondents and nonrespondents. For both 
surveys, it is very hard to quantify these 
differences because so little is known about the 
nonrespondents. Efforts to identify groups of 
households and adults that responded at 
different rates in the NHES (Brick and Broene 
forthcoming) did not reveal any major 
differences, but these efforts were limited by the 
lack of data on nonrespondents. 

Although it is not possible to obtain much 
information on differences between respondents 
and nonrespondents at the screening stage, some 
data are available on those sampled for the 
extended interviews. It is clear from the 
NHES:91 that response to the AE interview was 
related to whether or not the household 
respondent said the sampled adult was a 
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participant in adult education activities. In the 
NHES:91, the completion rate (the percent of 
sampled adults who completed the extended AE 
interview) for participants was 90 percent and 
the completion rate for nonparticipants was 82 
percent. In the NHES:95, the participant 
completion rate was 84 percent, while it was 77 
percent for nonparticipants. 

The difference in completion rates by 
participation status could result in a significant 
bias if nothing were done to address the 
differential bias. However, the differential 
response rates were recognized in the NHES:91 
and the NHES:95 and nonresponse adjustments 
were made to the weights to account for the 
difference and minimize the potential bias. No 
such adjustments were made in the NHES:95 
Splice sample because the participation status of 
the sampled adults was not asked in the 
Screener. In order to make this type of 
adjustment, data for both respondents and 
nonrespondents are needed, and if participation 
status is not collected in the Screener, this 
adjustment cannot be made. In the CPS, no 
special weighting adjustments were made for 
the supplement, but this may not have been as 
necessary due to the higher response rate in that 
study. 

To quantify the potential nonresponse bias, an 
example assuming a large difference between 
the characteristics of respondents and 
nonrespondents is explored to provide an upper 
bound on this source of bias. Assume that the 
percent of respondents that are participants is 40 
percent and the percent of nonrespondents that 
are participants is 20 percent. This is probably a 
large difference given the estimates from the 
various surveys. Applying equation (2) to these 
estimates with the NHES:95 nonresponse rate of 
40 percent results in an estimated bias of 8 
percent {40%(40%-20%)=8%}, implying that 
the participation rate from the NHES:95 could 
be as low as 32 percent (40% - 8 % =32%). The 
same approach can be applied to the CPS, 
assuming the same difference between 
respondents and nonrespondents, but with a 
nonresponse rate of 15 percent. This results in a 
bias in the CPS:92 estimate of 3 percent 




{15%(40%-20%)=3%}, implying an even lower 
participation rate of 21 percent (24% - 3% = 
21%). The lower bias in the CPS is due to the 
higher response rate and is the rationale for the 
statement above that special weighting 
adjustments to account for the different 
response rates of participants and 
nonparticipants may not be as important in the 
CPS. 

This exercise sheds some light on the potential 
for nonresponse bias in the estimates of the 
NHES and CPS. Estimates from both surveys 
may be biased, but the nonresponse bias in the 
NHES is probably larger than that in the CPS 
estimates. However, even assuming a very 
large difference between the characteristics of 
respondents and nonrespondents as was done in 
the example, the differential bias is probably 5. 
percent or less (8 percent in the NHES minus 3 
percent in the CPS). While differential bias 
may be a large contributing factor, this alone 
does not account for the large disparity in the 
estimates from the two surveys. 

Proxy Reports of Adult Education 
Participation 

Proxy reporting of adults' participation in AE 
activities is allowed for some purposes in each 
of the data sets under consideration in this 
report, with the exception of the NHES:95 
Splice interview. However, the surveys differ 
as to whether the proxy report is used only to 
select adults for interviewing or for classifying 
the AE participation status of the sampled adult. 
As noted above, in the CPS:84, a single 
household respondent reported on the 

participation status of all adults in the 
households and adults were selected for the AE 
supplement based on these proxy responses. 
Only those reported to be participants in the 
screening items were sampled for the 

supplement. The interview was conducted with 
the sampled' adult, but proxy respondents could 
also respond. In the CPS:92, participation items 
were asked about all adults, but proxy responses 
were accepted for adults who were not available 
for interviews. 
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In the NHES:91 and the NHES:95, a household 
respondent to the screener reported on the 
participation of all adults in the household; 
however, the sample for the AE interviews 
included both participants and nonparticipants. 
The participation status of the adults was 
determined by responses to the extended 
interview that could only be answered by the 
sampled adult. No proxy responses were 
allowed for extended interviews. In the 
NHES:95 Splice interview, one adult was 
sampled per household without first asking 
about participation, so proxy reporting was not 
involved. Only the sampled adult was eligible 
to answer the AE Splice interviews; proxy 
responses were not accepted. Thus, the use of 
proxy respondents in the NHES was very 
different from that in the CPS. 

Previous research on the reports of proxy 
respondents indicates that there may be some 
reporting error associated with this type of 
procedure (Moore 1988), but the results are not 
consistent or based on studies that permit valid 
estimation of the impact of proxy responses. In 
a study of reports of health status and 
satisfaction with medical care, Epstein et al. 
(1989) found differences between self and proxy 
reports, and found variation in the extent and 
direction of discrepancies based on such factors 
as the amount of time the proxy spent with the 
subject. In the field test of the NHES survey 
system, an analysis of proxy reports of the 
educational status of 14- to 21 -year-olds 
indicated that the correspondence of proxy 
reports and self reports of dropout rates varied 
according to the timing of the dropout event 
(e.g., whether it had occurred in the previous 
year) and the age of the youth (Mohadjer et al. 
1992). Thus, the issue of proxy reporting may 
be an important area to examine when assessing 
differences in AE participation rates. 

In the CPS:84, persons who were identified as 
nonparticipants at the screening stage were not 
selected for the AE supplement. As a result, it 
is not possible to compare proxy and self- 
reports as a possible source of error. 
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In the NHES:91, if the AE interview respondent 
was the screener respondent, the participation 
questions (which were the same in the screener 
and extended interview) were not repeated, so 
that the adult did not have the opportunity to 
change his or her status under these conditions. 
All of the "switching" in the NHES:91 
(described earlier) occurred when someone 
other than the adult sampled for the AE 
interview completed the screening items. In the 
NHES:95, a single brief screening item on 
participation in adult education was used for 
sampling purposes, and this item did not 
provide detail on what was included as adult 
education. As a result, the data collected in the 
NHES:95 does not support an analysis of the 
impact of proxy reporting on participation rates. 

Only the CPS:92 provides an opportunity to 
observe differences in participation rates based 
on proxy and self reports. In an assessment of 
methodological evidence concerning the quality 
of proxy reports in surveys, Moore (1988) 
questioned the adequacy of extant work, 
primarily on the basis of measurement problems 
and lack of random assignment to self report or 
proxy report status. The CPS:92 data indicate 
that self and proxy reporting does not occur 
randomly, supporting Moore's concerns. About 
58 percent of adults answered the AE items 
themselves, and 42 percent of cases had proxy 
respondents. In about 25 percent of proxy 
response cases, the proxy respondent was the 
spouse of the adult for whom he or she was 
reporting.^ 

Table 5 shows the percentages of adults for 
whom proxy responses were obtained for the 
adult education items, by several characteristics. 
Males, younger adults, Hispanic adults, those 
with more education, those who were employed, 
and those with higher incomes were more likely 
to have proxy respondents than females, older 
adults, non-Hispanic adults, those with less 
education, those not employed, and those with 
lower incomes. Because age, education, and 
labor force status are also associated with AE 



4 Special tabulations from the 1992 Current Population 
Survey (CPS). 
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participation (Korb, Chandler, and West 1991; 
Kim et al. 1995) these differences in rates of 
proxy reporting may confound any reporting 
error associated with the failure to obtain 
responses from the adult himself or herself. For 
example, higher rates of participation for those 



with proxy respondents could be related to 
differences in reporting by proxies, or to the fact 
that proxy reports are more likely for highly 
educated adults who participate in adult 
education at a higher rate than adults with less 
education. 



Table 5. — Number of sampled adults, by self/proxy reporting status and demographic characteristics: 
CPS:92 



Characteristics 


Number of self-reporters 


Number with proxy 
reporters 


Percent proxy reporting 


Total 


56,425 


40,645 


42 


Age 








Less than 30 years 


10,248 


11,415 


53 


30 to 49 years 


24,123 


16,420 


41 


50 years or older 


22,054 


11,810 


37 


Sex 








Male 


20,299 


24,837 


55 


Female 


36,126 


15,808 


30 


Race 

Black 


5,512 


3,475 


39 


Hispanic 


3,312 


3,009 


48 


Non-black, non-Hispanic 


47,601 


34,161 


42 


Highest education 
Less than high school diploma 


11,315 


6,713 


37 


High school diploma 


19,112 


15,875 


45 


Some college/vocational school 


13,345 


.11,157 


46 


Bachelor's degree or more 


12,653 


6,900 


35 


Employment status 








Not employed 


19,917 


16,403 


45 


Employed 


36,508 


24,242 


40 


Household income (1) 








Less than $10,000 


9,433 


3,616 


28 


$10,000 or more 


46,992 


37,029 


44 


Household income (2) 




. 




Less than $25,000 


25,714 


13,738 


35 


$25,001 to $50,000 


16,837 


13,632 


45 


More than $50,000 


10,468 


10,645 


50 



NOTE: Numbers equal the unweighted number of adults. 

SOURCE: Special tabulations from the 1992 Current Population Survey (CPS), Adult Education Supplement. 
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Table 6 shows the participation rates of adults 
who responded for themselves and those who 
had proxy respondents. The differences in 
participation rates for adults who responded for 
themselves and those with proxy respondents 
are small (3 to 4 percent). Self-reporters were 
slightly more likely to be participants than those 
with proxy respondents. However, as noted 



above, these small differences might be the 
result of the nonrandom chance of proxy status 
(persons for whom proxy respondents reported 
on AE participation may be more likely to be 
participants than nonparticipants) rather than the 
lack of difference between the two groups of 
adults. 



Table 6. — Percent of adults participating in adult education activities, by source of report: 1992 







Percent participating in 


Participation status 


Number 
(in thousands) 


Any activity 


Any activity excluding full- 
time college 


Any activity 
excluding college 


All adults 


169,722 


24.0 


20.5 


17.0 . 


Self reporter 


97,947 


24.0 


21.7 


18.8 


Proxy reporter 


71,758 


24.0 


18.9 


14.5 



NOTE: Excludes about 8 percent of the sample, for whom the source of the report (self/proxy) was not identified. 
SOURCE: Special tabulations from the 1992 Current Population Survey (CPS), Adult Education Supplement. 



To examine the extent to which estimates of 
participation are affected by differences in the 
characteristics of self-reports arid proxy reports, 
a logistic regression analysis was conducted. In 
the regression models, the variable being 
predicted was participation status (either a 
participant or not a participant). The predictor 
variables were those related to either 

participation status or self/proxy status. In 
addition to the predictor variables, an indicator 
variable for proxy status (1 if a proxy and zero 
if not) was included in the models. Table 7 
shows the predictor variables in the logistic 
regression model where the variable being 
predicted was participation in any type of adult 
education activity, excluding full-time college 
enrollment. 

The first step was to evaluate whether or not 
proxy status was an important predictor for AE 
participation when the other predictor variables 
were included in the models.. For all three types 
of adult education (any activity, excluding full- 
time college, and excluding any college 
activity), the proxy indicator was statistically 



significant, confirming the importance of proxy 
response status iri estimatirig AE participation. 

The regression analysis confirms the importance 
of proxy reporting and suggests that the small 
differences in participation rates in table 6 may 
be due to the nonrandom assignment of proxy 
reporting ' status. To examine this more 
carefully and quantify the proxy reporting 
effect, the parameter estimates from the logistic 
regression models can be used to estimate what 
the participation rate would have been in the 
CPS:92 if all adults had reported for themselves. 
This is done in two steps. First, the probability 
of participating in adult education was estimated 
for each sampled adult in the CPS:92 using the 
estimated regression parameters given in table 
7, but the self/proxy indicator was always set to 
self. This resulted in an estimated probability of 
participation if all adults self-reported. Second, 
the estimated probabilities were summed over 
all the sampled adults using the CPS:92 
estimator weights to produce a predicted 
participation rate. 
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Table 7. — Logistic regression parameters for a main effects model of AE participation, excluding full 
time degree programs 



Model parameter 


Parameter estimate 


Standard error 


Intercept 


1.82 


.04 


Whether self-reported or proxy-reported 


-.28 


.02 


Whether had a high school diploma or not 


1.36 


.04 


High school diploma 


.96 


.02 


Some college or associate’s degree 


.33 


.02 


Whether employed or not 


-.41 


.02 


Family income under $25,000 


.51 


.02 


Family income between $25,000 and $49,999 


.20 


.02 


Gender 


-.06 


.02 


Age 16 to 25 years old 


-1.04 


. .03 


Age 26 to 55 years old 


-.95 


.03 



SOURCE: Special tabulations from the 1992 Current Population Survey (CPS), Adult Education Supplement. 



Table 8 shows the observed participation rates 
and the rates predicted if all adults had reported 
for themselves. The participation rate for any 
activity excluding full-time college is estimated 
to be 2.2 percent greater if all the adults had 
reported for themselves (22.7 - 20.5 = 2.2). For 
the other types of participation, the rates 
increase nearly 3 percent, using the same 
modeling procedures. 

Thus, these results indicate that proxy reporting 
in the CPS:92 probably resulted in a bias of 



nearly 11 percent in the percent of adults who 
participated in adult education activities (2.7 
percent is 11.3 percent of the observed 
participation rate of 24.0 percent for activities 
excluding full-time college). If all adults had 
responded for themselves, it is estimated that 
the participation rate for any AE activity would 
be about 27 percent. While this is a significant 
source of error, proxy reporting accounts for a 
relatively small part of the difference between 
the CPS and NHES estimates. 



Table 8.— Observed and predicted participation rates if all adults had self-reported, by type of 
participation: CPS:92 



Type of participation 


Observed participation 
rate 


Predicted self-reported 
participation rate 


Any adult education activity 


24.0 


26.7 


Activity excluding full-time college 


20.5 


22.7 


Activity excluding college 


17.0 


20.1 



SOURCE: Special tabulations from the 1992 Current Population Survey (CPS), Adult Education Supplement 
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Survey Context 

The context in which a survey takes place can 
have a substantial impact on the resulting 
estimates. Many readers may think of survey 
context effects in terms of question order; 
indeed this is related to the issues we address. 
But this effort takes a broader view of the 
context in which the survey takes place. In this 
section, we address three aspects of survey 
context that could have an effect on estimates of 
AE participation: questionnaire wording, 

sponsorship, and interview supplement status. 
Each of these aspects of the survey create a 
context that may affect the interviewers’ and 
respondents’ perception of the content and 
import of the survey. 

Two other aspects of survey context were 
considered, but are not addressed in detail here, 
because we do not believe that they have an 
impact on estimates of AE participation. The 
first of these is the mode of administration. 
While mode effects have been documented in 
the literature (Sudman and Bradbum 1974; 
Bradbum 1983; Dillman and Tamai 1991), this 
does not seem to be a plausible source of 
differences in the NHES and CPS estimates of 
participation. The majority of CPS interviews 
are conducted by telephone, which is the only 
NHES mode of administration. A second area 
that was considered is the timing of the surveys. 
The NHES AE components were conducted 
during the winter and early spring months, 
where as the earlier triennial CPS AE 
supplements were conducted in May, and the 
CPS:92 was conducted in October. The time of 
year might have had an important impact on 
recall if AE activities were tied to a traditional 
school year, and this could be used to prompt 
recall. However, many AE activities are not 
structured around a traditional school year, and 
all surveys under consideration had 12-month 
recall periods. 

Questionnaire Wording 

The way questions are asked in a survey can have 
a major impact on responses, as is well 
documented in the literature (Groves 1989; 



Converse and Presser 1986; Schwarz and Hippier 
1991; Kalton and Schuman 1982). Therefore, it 
is reasonable to expect that the way in which a 
question about AE participation is asked could 
have an impact on estimates of participation. 
Converse and Presser (1986; p. 18) note the 
considerable difficulty inherent in attempting to 
convey a frame of reference to a respondent and 
having the respondent adhere to that frame of 
reference in the interview. Questionnaire items 
on participation can attempt to convey the frame 
of reference in a number of ways, for example: 

■ provide examples or definitions that 
clarify the concept of adult education and 
what it includes; 

■ set exclusionary criteria to prevent "false 
positives"; and 

■ provide time parameters that assist the 
respondents' recall. 

In each of the data collections discussed in this 
report, specific questions were asked that were 
designed to provide the respondent with prompts 
and recall aids so that all appropriate educational 
activities would be reported. However, the 
questions were not the same in each of the data 
collections. The specific questions for each of 
the five data sources appear in appendix A. This 
section provides a summary of wording 
differences and interpretation of some of the 
implementation. Details about how adult 
education was defined and how parameters affect 
estimates of participation rates will be discussed 
later. 

In the CPS:84, the NHES:91, the CPS:92, and the 
NHES:95 Splice interview, the words "adult 
education" were used in at least some of the 
questions or to introduce a series of questions. 
Cognitive laboratory work conducted for the 
NHES:95 indicated that the phrase "adult 
education" has different meanings for different 
people. Some persons think of formal 
instruction, and some think of classroom settings 
only. Many adults exclude courses, classes, or 
lessons taken for recreational purposes when 
thinking about adult education. While a list of 
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specific activities followed the words "adult 
education," some respondents may have been 
focusing on their own conception of "adult 
education" when responding to the items. An 
intensive reinterview Bias Study conducted for 
the NHES:95 supports this notion, and suggests 
that respondents may form their own paradigm as 
to what is included in the survey very early in the 
interview, especially when the initial questions 
focus on "traditional" classroom activities such as 
basic skills programs, English as a Second 
Language (ESL) programs, and postsecondary 
credential programs, as in the NHES:95 (Brick et 
al. 1996). 

Defining the Parameters of Adult Education. 

Different approaches were used in the various 
data collections to inform respondents about the 
nature of the interview and set up the response 
paradigm. Often this was done by providing the 
adults with inclusionary and exclusionary 
information on adult education. In the CPS:84, a 
flashcard was provided that gave a definition of 
adult education, and provided examples of types 
of activities or courses that would or would not 
be included in the definition. The examples 
given in the flashcard did not specifically 
mention English as a Second Language (ESL) or 
GED preparation courses, which were explicitly 
included in the other surveys in this analysis, 
although these activities could have been 
included by respondents under categories such as: 
“all adult and continuing education and noncredit 
courses or activities,” or “basic instruction for 
adults who have never attended school or who 
have interrupted their formal schooling at the 
elementary or secondary level.” It is difficult to 
know exactly how this flashcard was used, 
because 70 percent of the interviews were 
conducted by telephone and therefore most 
respondents did not see the flashcard. 
Interviewers were instructed to read or 
summarize the information on the card for the 
respondents. However, because the amount of 
information contained on the card was fairly 
extensive, many respondents may not have 
absorbed and retained all of the information, even 
if the interviewers did read it. In addition, it is 
not known how interviewers were trained to 
summarize the card. 



In the CPS:92, the questions were included as a 
part of the October education supplement, and 
no AE-specific introduction was provided. This 
could be partly responsible for the large 
differences between the NHES and the CPS:92 
estimates. 

The introduction in the NHES:91 and the 
NHES:95 Splice interviews was very different 
from that of the CPS:84 and the NHES:95 AE 
interview. In the NHES:91 and the NHES:95 
Splice sample, respondents were told that the 
survey covered "three kinds of educational 
activities over the past year: full-time school, 

part-time school, and all other kinds of adult 
education." Respondents were also informed that 
educational activities for job-related purposes, 
personal enjoyment, or recreation were included. 
No detailed information was provided prior to the 
specific questions about AE participation. 
Because these items were designed specifically 
for telephone administration, lengthy 
introductory explanations were avoided. 

In the NHES:95 interview, the introductory 
approach was substantially modified based on 
experience from previous data collections and the 
results of cognitive laboratory work and 
telephone field testing. Adult education was not 
specifically mentioned in the introduction. 
Rather, the questionnaire was structured so as to 
have a section on each type of adult education. 
Each type of activity was asked about separately, 
with appropriate explanation or examples. Thus, 
each section defined a type of activity and 
provided examples or exclusions regarding that 
specific type of activity. (The specific items 
appear in appendix A.) 

The different NHES approaches to introducing 
the adult education items did not result in 
differences in participation rates between the 
NHES:95 AE and the Splice sample (which was 
the same as the NHES:91). 

Exact Wording. In their work on the 
disparities in training estimates, Zemsky and 
Shapiro (1993) note that items from different 
surveys often use different wording and formats 
to collect information on participation. The use 
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of different items in various data sources, 
including the National Longitudinal Survey of 
Youth, the Survey of Income and Program 
Participation, the triennial CPS surveys, and 
High School and Beyond, present the researcher 
with a considerable challenge in developing a 
systematic measure of participation. A similar 
challenge is found in comparing CPS and NHES 
items. 

In addition to differences in the introductions and 
definitions of activities in the CPS and NHES 
interviews, there were also differences in the 
specific activities that were included. The 
NHES:95 differs significantly from the CPS:84, 
the NHES:91, the CPS:92, and the NHES:95 
Splice interviews, all of which are quite similar to 
one another. For example, the CPS: 84 included 
courses taken by "correspondence, radio, 
television, or newspaper...." The NHES:91, 
CPS:92, and NHES:95 Splice interviews included 
"courses by mail, television, radio, or 
newspaper." Also, all of the instruments except 
the NHES:95 AE interview asked about 
continuing education courses or noncredit 
courses; private instruction or tutoring; education 
or training given by employers, labor 
organizations, neighborhood centers, churches, or 
community groups; or any other "organized 
educational activity." While the CPS:84 asked 
about courses for adults who had not finished 
high school, the NHES:91, CPS:92, and 
NHES:95 Splice interviews asked about 
instruction in basic skills such as math and 
English, and also asked about instruction in 
English as a Second Language. 

In the NHES:95, items and questionnaire sections 
focusing on six types of AE were developed: 
English as a Second Language; basic skills 
including adult basic education (ABE) and 
preparation for the General Educational 
Development (GED) examination; credential 
programs to earn college degrees or vocational, 
technical, or occupational certificates or 
diplomas; apprenticeship programs; work-related 
courses or training; and other formal structured 
courses, such as personal development, 
recreation, and so on. The NHES:95 did not 
specifically ask about correspondence courses or 
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those taken by mail, radio, television, or 
newspaper, and the section on other formal 
structured courses (also called personal 
development courses) specifically asked for 
courses "where there was an instructor." 

The new approach in the NHES:95 AE interview 
created a concern about the impact restructuring 
of the instrument and changes in wording might 
have on estimates of participation. The NHES:95 
Splice interview was implemented primarily to 
compare estimates using the NHES:91 questions 
in the Splice interview and the new NHES:95 AE 
questions in a concurrent data collection. This 
permitted an assessment of whether any change 
in participation rates could be attributed to 
changes in the items themselves, and the extent of 
the instrument effect on the estimates. In fact, 
however, the participation rates for the NHES:95 
AE interview and the NHES:95 Splice interview 
were virtually identical -- 40.2 percent and 40.0 
percent. 

Because the NHES:95 items and the NHES:95 
Splice items were collected at the same time 
using the same survey procedures and yielded 
nearly identical estimates of participation, it 
appears unreasonable to attribute the large 
differences between CPS and NHES estimates to 
differences in item wording. Also, this evidence 
suggests that the apparent increase in adult 
education between the NHES:91 and the 
NHES:95 should not be attributed to wording 
differences in items concerning participation. 
However, there are other factors related to the 
survey context that must be considered. 

Interview Status: Sponsorship and 
Survey Focus 

Two other context differences between the 
NHES and the CPS are so closely related as to 
be inseparable — sponsorship of the survey and 
the primary focus of the survey activity. In the 
NHES, the respondents are told that this is an 
interview conducted for the U.S. Department of 
Education and is related to education issues. 
The CPS is a survey conducted by the Bureau of 
the Census for the U.S. Department of Labor to 
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estimate characteristics of the labor force. The 
questions about adult education are only 
included in a supplement to the CPS and are not 
the main focus of the study. Thus, the overall 
context of the interviews with adults about 
education are different from the perspective of 
both the respondent and the interviewer. 

These differences may manifest themselves in 
numerous ways. For example, the NHES 
interviewers undergo 16 hours of specific 
training to conduct the NHES interviews, 
including practice interviews under the direct 
supervision of trainers and telephone center 
staff. The training for supplements in the CPS 
is minimal, most often consisting of a home- 
study package that the interviewers are 
supposed to read. The level of training can 
make an important difference in the quality of 
the interviews, as suggested by Brick et al. 
(1995) in a comparison of interviewer effects in 
two surveys. Perhaps the training or lack of 
training sends a more direct message to the 
interviewers about the importance of the 
interview. If the interview is a supplement with 
little or no training required, then the 
interviewers may perceive the data to be less 
important. 

It is possible to speculate on how the overall 
context of the survey might affect the responses 
to the survey, but these speculations cannot be 
quantified due to the lack of data. For example, 
it seems likely the responses to standard items, 
such as college enrollment, that are asked every 
year in the March CPS, would be more reliable 
than the questions about other types of adult 
education. In fact, the AE Bias Study (Brick et 
al. 1996) showed that even in the NHES:95 
many of the adults under-reported less formal 
types of adult education activity (in particular, 
work-related and personal development 
courses). It is very likely that this type of 
under-reporting was even more prevalent in the 
CPS for the reasons discussed above. 

As suggested above, the status of a survey as 
being a supplement to another survey could 
have an effect on the quality of the data being 
collected. Although no references in the 



literature were found that addressed this specific 
issue, it is interesting to note that Shapiro (1987) 
did investigate the opposite: the effect a 

supplement might have on the estimates from 
the core study. In a review of four surveys, he 
found that the core estimates were affected 
when supplementary questions were asked later 
in the interview (after the core items were 
collected). He speculated that the observed 
differences might be the result of the 
interviewer’s anticipation of the supplemental 
questions that might alter the probing that is 
done in the core study. 

Shapiro’s results suggest that it is very possible 
that the supplement estimates may also be 
affected by the core study, even though he did 
not examine this situation. For example, it is 
possible that the interviewer and/or respondent 
may place less attention on the items in a 
supplement than on the core items, leading to 
less thoughtful administration and response to 
the supplementary items. Although it is not 
possible to quantify the difference without a 
follow-up or reinterview study to the CPS 
specifically designed for this purpose, it is 
feasible that the context of the survey as a 
supplement could have resulted in under- 
reporting of participation of adult education in 
the CPS. 

Findings of the NHES:95 Bias Study cited 
above (Brick et al. 1996) and the supplement 
status of the CPS items on AE participation may 
interact to produce the observed estimates. The 
Bias Study results suggest that the reporting of 
adult education activities, particularly those that 
fall outside of formal schooling, is a cognitively 
demanding task. It may be that respondents are 
more successful at rising to this task in a survey 
that focuses specifically on adult education, as 
compared to one in which adult education is 
a 

supplementary issue. The CPS has a long 
history of successfully collecting information on 
formal schooling, which appears from the 
NHES:95 findings to be a less demanding task 
for the respondent. 



Summary 

Reliable estimates of participation in adult 
education are important for a number of 
reasons. One of the key roles of these 
participation rates is to inform the public about 
the National Educational Goal (National 
Education Goals Panel 1994), which states that 
“By the year 2000, every adult American will be 
literate and will possess the knowledge and 
skills necessary to compete in a global economy 
and exercise the rights and responsibilities of 
citizenship.” Participating in various forms of 
adult education is an activity that may support 
achieving this objective. 

The focus of this study is an assessment of 
reasons that estimates from the NHES and CPS, 
two important studies that measure adult 
education participation, are so different. As 
noted in table 1 , the estimates from the CPS are 
substantially lower than those from the NHES. 
The report examined a number of 
methodological, definitional, and operational 
factors that might have had an impact on the 
differences in the estimates. Some of these 
factors could be quantified so as to estimate the 
potential impact they had on the estimates, 
while other factors evade measurement ^iven 
the available data. Relevant findings are 
summarized in table 9 and discussed below. 

The 1984 CPS estimate of participation in adult 
education excluding full-time college was 14 
percent and for 1992 was 21 percent. An 
increase of 7 percent in adult education over the 
8 years is reasonable given the efforts made by 
government and business to increase the skills 
of adults during this time. However, the two 
CPS studies used very different approaches (in 
particular, different approaches to selecting 
adults) that should have had a substantial 
influence on the participation estimates. As 



explained earlier, if the CPS: 84 had sampled all 
adults regardless of the responses of the 
household respondent, it is estimated that the 
‘switching rate’ would have caused the 
participation rate in 1984 to be 23 percent. This 
is close to 26 percent adjusted estimate from the 
CPS:92. This very modest increase in 

participation rates in adult education between 
1984 and 1992 is not consistent with what 
experts in adult education expect (Kim et al. 
1996), nor is it consistent with the higher rates 
estimated in the NHES. This raises some 
questions about the validity of the CPS:92 
estimates. 

Comparing the CPS and the NHES estimates, a 
number of factors were considered. One of the 
most obvious factors is the bias in the NHES 
estimates due to the failure to sample adults in 
households without telephones. However, the 
empirical evidence clearly indicates that the 
undercoverage bias in the NHES estimates is 
very small (less than one percent), especially 
when compared to the differences between the 
CPS and NHES estimates that range from 1 3 to 
20 percent. 

A second methodological difference is the 
estimated bias due to nonresponse that affects 
both surveys, but has a larger impact on the 
NHES because of the lower response rate in this 
study. The nonresponse biases could not be 
estimated directly because of the lack of data on 
the differences between respondents and 
nonrespondents to the surveys. Nevertheless, by 
making some assumptions about these 
differences, an upper bound on the differential 
nonresponse bias was estimated. The bound 
shows that the NHES may overestimate the 
participation rate by up to 5 percent more than 
the CPS. This is a potentially important source 
of bias, but it is an upper bound and the actual 
bias due to nonresponse is probably less than 5 
percent. 
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Table 9. — Summary of factors associated with possible biases in estimates of adult education 
participation, by survey and possible size of bias 



Factor 


Survey 


Possible size of bias' 


Sampling only participants 


CPS:84 


-9% 


Sampling only telephone households 


NHES:91 , NHES:95, and Splice sample 


<1% 


Unit nonresponse 


CPS:84 and CPS:92 


<3% 2 




NHES:91 , NHES:95, and Splice sample 


<8% 2 


Proxy respondents 


CPS:84 and CPS:92 


-3% 3 


Context effects 






Introductory statements 


All surveys 


No measurable effect 


Wording 


All surveys 


No measurable effect 


Supplemental status 


All surveys 


Unmeasurable 4 


Interviewer training 


All surveys 


Unmeasurable 4 


Survey sponsorship 


All surveys 


Unmeasurable 4 



'Bias is the difference between the survey estimate and expected population value. A negative bias indicates the survey estimate 
is less than the population value. 

^Biases estimated for the CPS:92 and NHES:95. 

3 Bias estimated for the CPS:92. 

‘' these context effects can not be estimated from the available data, but could be substantial. 

SOURCES: U.S. Department of Commerce, Bureau of Census, Current Population Survey (CPS), 1984 Adult Education 

Supplement, 1992 Adult Education Supplement; U.S. Department of Education, National Center for Education Statistics, National 
Household Education Survey (NHES), spring 1991, spring 1995, spring 1995 Splice Sample Interview. 



A third potential source of error was the use of 
proxy respondents, particularly the practice in 
the CPS where other adults could respond for 
the sampled adult. In the NHES, the sampled 
adult was the only person who could complete 
the extended interview and the responses of 
other adults in the household were only used for 
sampling purposes. The use of proxy 
respondents was found to be an important 
predictor of AE participation of the adults in the 
sample, with those adults who responded 
themselves being more likely to participate. By 
using logistic regression analysis, the potential 
impact of proxy reporting was estimated for the 
CPS:92. These findings show the estimates 
from the CPS:92 are biased downward by about 
3 percent. No corresponding bias exists in the 



NHES estimates, since proxy reporting was not 
permitted. 

The next aspect of the surveys considered was 
the overall context of the surveys. As noted, the 
literature on context effects is diverse, and in 
some cases context effects can be substantial. 
The first two aspects of context that were 
considered were the introductory approach that 
sets the groundwork for the survey and 
questionnaire wording. The NHES:95 Splice 
sample, a methodological study especially 
designed and implemented to address these 
issues, reveals that these two factors are not a 
source of the differences between the NHES and 
CPS estimates. 
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Other context issues were considered but could 
not be quantified. One of these was the fact that 
the adult education questions in the CPS were a 
supplement to the core CPS survey. This 
clearly resulted in different operational 
approaches that may have influenced the 
interviewers and the respondents. The training 
of interviewers for the adult education 
interviews was also obviously different. The 
question is whether this affected the estimates 
and, if so, by how much. In addition, 
supplemental status may result in the questions 
holding a place of lesser importance or focus in 
the minds of respondents, and perhaps 
interviewers. The second related issue is the 
sponsorship of the study. Sponsorship has been 
shown to have an influence on the responses to 
a survey, but again this cannot be quantified for 
these studies. 

Putting these sources of error together and 
assuming the errors are additive, much of the 
difference between the NHES and CPS 
estimates of participation could be explained. 
For example, the NHES:91 estimate of 33 
percent may be an overestimate by as much as 5 
percent due to undercoverage and nonresponse 
bias, while the CPS:92 estimate of 21 percent 
may be an underestimate by as much as 3 
percent due to proxy reporting and nonresponse. 
The remaining difference of 4 percent (28% - 
24% = 4%) could easily be accounted for by 
context effects. 

While this approach does account for much of 
the difference, it is not a very satisfying solution 
for several reasons. First, the CPS:92 estimates 
appear to be out of line with the other estimates, 
including the CPS:84 estimates. As noted 
above, the CPS:84 estimates adjusted for not 
sampling nonparticipants who might have 
participated results in an estimate of 23 percent, 
only very slightly lower than the 26 percent 
adjusted CPS:92 estimate. Second, the 
differential nonresponse bias in the two surveys 
is really an upper bound and the difference 
between the NHES and CPS due to this source 
is probably much less than 5 percent. Third, it 
is unlikely that the error sources are actually 
completely additive. The errors due to 



coverage, nonresponse, use of proxy 
respondents, and context effects almost 
certainly interact in some complex ways. 
However, approximating the influence of these 
effects as additive is all that can be done, given 
the data, and appears to be a reasonable 
approach. 

The evidence that most conflicts with this 
simple approach to summing the errors is the 
Bias Study conducted in the NHES:95 (Brick et 
al. 1996). All of the sampled participants 
(based on the original NHES:95 interview) who 
were included in that intensive reinterview 
study proved to be participants, while 41 percent 
of the sampled nonparticipants reported an adult 
education activity that they had not reported in 
the original interview. The activities reported 
by these nonparticipants were examined and did 
appear to be real activities undertaken during 
the time frame of the survey. The findings in 
Bias Study strongly suggest that even the 
NHES:95 estimates, which are larger than any 
previous estimates of participation, may still 
underestimate participation. That study 
suggests that the estimates of participation rates 
might be substantially higher than 40 percent. 

These issues make it difficult to arrive at a 
conclusion without some ambiguity, although 
some points appear to be based on solid 
foundation. The NHES estimates are probably 
subject to some bias due to nonresponse, but 
there is little bias coming from telephone 
coverage bias. The evidence from the Bias 
Study indicates that respondents are 
underreporting, so the overall estimates of 
participation in adult education are probably 
even larger than the estimates produced from 
the NHES. The CPS estimates from 1984 and 
earlier were biased downward due to several 
sources, most notably the failure to sample 
adults classified as nonparticipants by the 
household respondent, proxy response, and 
context effects associated with the survey being 
a supplement to a labor force survey. The 
underreporting of activities noted in the Bias 
Study for the NHES:95 are probably even 
greater in the CPS due to the lack of training 
with respect to the adult education supplement. 
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Although the results of this study are subject to 
other interpretations, a reasonable conclusion is 
that the NHES estimates of participation are 
more accurate representations of the percentage 
of adults who participate in some type of adult 
education than the CPS estimates. As the Bias 
Study suggests, the participation rate is likely to 
be even higher than the NHES estimates due to 
respondent underreporting activities. The CPS 
estimates are depressed by the series of factors 
noted above. Furthermore, the Splice sample 
shows that change in the participation rate 
between 1991 and 1995 is a reflection of an 
actual change in the rates and not an artifact of 
the survey methodology. 

Since the results of this study are not definitive, 
it is interesting to consider options that could be 
used to further investigate the differences 
between the NHES and CPS estimates if future 
studies were to be conducted. A study of the 
CPS might be most useful to test the hypothesis 
that the CPS is severely biased downward. To 
conduct such a study, another supplement to the 
CPS could be conducted using the same 
approach used in 1992. An intensive 
reinterview, similar to the NHES Bias Study, 
could then be conducted with a sample of the 
respondents (both proxy and sampled adults) to 
ascertain the size of the bias in the estimates. 
This approach is rather expensive and should 
only be considered if future adult education 
surveys might be conducted as supplements to 
the CPS. 



The biggest potential source of error in the 
NHES, other than respondent underreporting 
which has already been at least partially 
examined in the Bias Study, is unit nonresponse. 
This source of error is a significant problem for 
all NHES administrations, and for that matter all 
telephone surveys, because telephone surveys 
often result in lower response rates than in- 
person surveys. A number of methods could be 
used to address this concern. First and 
foremost, the importance of conducting the 
surveys using techniques to increase the 
response rates should always be considered. 
Given the best efforts to improve response rates, 
studies of nonresponse might be undertaken. 
Different study designs should be considered, 
such as record check studies, where appropriate 
and feasible. Follow-up studies could also be 
undertaken, but these studies tend to be 
expensive, especially if they involve matching 
telephone numbers to addresses and conducting 
in-person interviews. Less expensive methods, 
such as matching telephone numbers to files 
with demographic data on households, are 
feasible, but limited because the data in the files 
is often of poor quality and a large percentage of 
the telephone numbers may not be matched 
correctly. Despite these difficulties, some 
information from these studies might improve 
the understanding of the nature and size of 
nonresponse bias in telephone surveys. 
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Appendix A 



Adult Education Participation Items 



This appendix presents the adult education participation questions that were used in 1984 Current 
Population Survey Adult Education Supplement (CPS:84), in the 1991 National Household Education 
Survey Adult Education component (NHES:91), in the 1992 Current Population Survey Adult Education 
Supplement (CPS:92), in the 1995 National Household Education Survey Adult Education component 
(NHES:95), and in the 1995 National Household Education Survey Splice Sample Adult Education 
interview. 
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CPS:84 Adult Education Screening Items 



31. Is ... attending or enrolled in a high school, college, or university? 

Yes (verify: High school, College or Univ.) 

No 

32. Is ... enrolled as a full-time or part-time students? 

Full-time 

Part-time 

33. Is ... now attending or enrolled as a full-time students in a vocational or occupational program? 

Yes No 

34. Now I would like to ask some questions about ...'s participation in adult education activities. 
(Read or show definition and list of examples on flashcard.) 

(Excluding full-time school attendance) during the past 12 months (that is, since May 1 a year 
ago) has ... (Ask and mark each category) 

a. Taken any adult or continuing education or noncredit courses or educational activities? 

Yes No DK 

b. Taken any courses for credit as a part-time students in high school, college, vocational, or 
other school? 

Yes No DK 

c. Taken a course by correspondence, television, radio, or newspaper, or taken any private 
instruction or tutoring? 

Yes No DK 

d. Taken any courses or educational activities given by an employer, a labor organization, 
neighborhood center, a church, or another community group? 

Yes No DK 

e. Taken any instruction for adults who have not finished high school? 

Yes No DK 

f. Taken any other organized educational activities or courses during the past 12 months? 

Yes No DK 
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CPS:84 Adult Education Supplement 
Participation Items 



Are you now attending or enrolled as a full-time student in a school of any type? 

Yes (Ask 2) No (Skip to 5) 



Are you in a program leading toward the completion of elementary school or high school? 

Yes (Skip to 6) No 

Are you in a college degree program (such as an associate’s, bachelor’s, master’s, or doctoral 
degree program)? 



Yes (Skip to 6) No 

Are you in a vocational or occupational program? 

Yes (Skip to 6) No (Skip to 6) 

Are you now enrolled in a college degree program or university as a part-time student, in courses 
for credit toward a degree (such as an associate’s, bachelor’s, master’s, or doctoral degree 
program)? 



Yes No 

(Excluding full-time attendance in school) during the past 12 months, that is since May 1, 1983, 
have you taken part in any organized adult education courses or activities such as ....? 

(Read or show definition and list of examples on flashcard.) 

Yes (Ask 7) No (End questions for this person) 

How many of these courses or activities did you participate in, including any that you did not 
complete? 



